← All Tags
# inference → benchmarks → on device llm → bonsai 8b

1 link tagged with all of: inference + benchmarks + on device llm + bonsai 8b

Click any tag below to further narrow down your results

+ 1 bit quantization (1)

Links

I Tested the 1-Bit LLM That Fits in 1 GB — It Shouldn't Be This Good | by Chew Loong Nian | in Level Up Coding

PrismML’s Bonsai 8B trains a large language model with 1-bit weights from scratch, squeezing 8.2 billion parameters into just 1.15 GB. In benchmarks it ties or outperforms FP16 models like Llama 3.1 and runs at real-time speeds on phones, shifting the size-performance trade-off.
  • Bonsai 8B packs 8.2B parameters into 1.15GB using native 1-bit weights, yet scores 70.5 average vs Llama 3.1's 67.1 (16GB FP16), even hitting 88.0 on GSM8K vs Llama's 76.6.
  • It runs at 44 tokens/sec on an iPhone 17 Pro Max, making full on-device 8B-scale LLMs feasible without cloud infrastructure.
  • Its "intelligence density" (score/GB) hits 1.062 versus Qwen's 0.098 and Llama's 0.084 — over 10x more capability per byte.
  • Trade-offs appear in code generation (57.9 vs Qwen's 79.9 on HumanEval+) and multi-step reasoning (MuSR: 64.3 vs 70.0), showing 1-bit precision struggles with complex logical chains.
Last saved Apr 25, 2026 · 6 min read
bonsai 8b + 1 bit quantization on device llm inference benchmarks